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Abstract 

Between January and April 2000, Mount Royal College in Calgary, 

Alberta piloted an anonymous online faculty -administered teaching 
assessment tool. The quantitative and qualitative results from the pilot 
are discussed and used to determine what influence, if any, the tool 
had on the teaching and learning environment. Preliminary findings 
suggest that the software was valuable from both student and faculty 
perspectives. 

Introduction 

Teaching appraisals are like a compass on a ship: without one, no one has a sense of 
direction— all hands are lost. A student’s assessment of a teacher is always subjective, at 
times unfair, and, possibly, stressful, but it is one of the few instruments to indicate if we 
are about to sail off the edge of the world or discover a new continent. 

Traditionally, teaching appraisals are conducted at the end of term. However, this 
prevents students from offering constructive feedback while they are still in the course. 
Waiting until the end of the term indicates that student input is valued only after a course 
is over, which implies that it is not valued at all. Conducting ongoing teaching 
assessments, on the other hand, opens a proactive dialogue with students about teaching, 
the course, and the learning process. 

The ongoing assessment of a course and/or the teacher is a critical and fimdamental 
component of the learning process. In the following paper, I comment briefly on the 
student assessment literature and then review a research initiative conducted at Mount 
Royal College on the application of an online assessment tool. The discussion includes 
an analysis of the merits of the tool from both student and faculty perspectives. 

Literature Review 

The validity and reliability of student ratings of teaching has been thoroughly analyzed 
and generally supported in the literature (see Ali and Sell, 1998; Chen and Hoshower, 
1998; Cohen, 1980, 1981; Feldman, 1989; Jackson, et al., 1999; Kwan, 1999; Marsh, 
1987; Marsh and Roche, 1997; McKone, 1999; Wachtel, 1998; Young and Shaw, 1999). 
Even with this support there continues to be much debate about if, how, when and in 
what form students should be allowed to assess their instructors (CAUT, 1998). A 
review of the vast literature on student assessment of teaching is beyond the limitations of 
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this paper, however, a look at a few key issues around student's assessment of instructors 
should prove useful for situating our discussion.^ 

Kam-Por Kwan's (1999) research into student teaching assessments was based on a 
comparative analysis of 4,996 courses (across 25 departments) at a university in Hong 
Kong. His findings suggest that there are significant differences in student ratings across 
academic disciplines (applied languages and design scored highest while applied sciences 
and engineering scored lowest) and that academic discipline explains about 10% of the 
total variance in students ratings (Kwan, 1999: 191). Kwan also found that class size 
accounted for 15% of student rating variations - smaller classes tended to receive higher 
ratings than larger classes. Although Kwan's research findings are revealing, he notes 
that his research does not attempt to explain why these differences in student ratings 
exist. His findings suggest that there are substantive differences in student ratings across 
academic disciplines and instructional settings but that these are generally outside the 
instructor's direct control and thus, student assessments should not be used as the sole 
means for evaluating individual instructional abilities (Kwan, 1999:193). 

Yining Chen and Leon B. Hoshower (1998) applied expectancy theory to an investigation 
of the factors motivating students to participate in the teaching evaluation process. They 
found that students felt that the most attractive outcome of teaching assessments was to 
improve teaching. Students were also motivated to participate if they believed their 
comments would be used to improve the course curriculum or if other students were 
going to see their comments. From the student's perspective the least important use for 
teaching assessments was for a teacher's tenure, promotion or salary increments (Chen 
and Hoshower, 1998). Their research also confirmed that student's motivation to 
participate in teaching assessments improved if students felt they were being asked to 
provide meaningful feedback and that it would be acted upon. Chen and Hoshower's 
work is useful for revealing why students may, or may not, participate in completing a 
teaching assessment. Faculty's perspective on student's teaching assessments is quite 
different however. 

Faculty concerns about student assessments of their teaching generally fall into three 
camps. First, some argue that because teaching is a personal endeavour that is influenced 
by one's personality and beyond the ability of standardized forms to assess in any 
meaningful way (see CAUT, 1998; Omstein, 1990). Second, because teaching 
evaluations are standardized, faculty may be less motivated to experiment in their 
instructional methods (Centra, 1993). The fact that some faculty may teach to the 
evaluation reduces faculty morale and job satisfaction and may be at odds with good 
educational practice (Wachtel, 1998). Third, the use of anonymous student assessments 
of teaching suggests that students can vent off steam without ramifications or 
accountability (Marlin, 1987 as cited in Wachtel, 1998:204; see also CAUT, 1998). 




^ For excellent reviews of the research into student's assessment of teaching, see Ali and 
Sell, 1998; Wachtel, 1998; Young 8c Shaw, 1999. 



3 



Given these concerns there is clearly more work to be done to make student assessments 
of teaching a more friendly and constructive mechanism for improving teaching. An 
ideal student assessment tool would allow students and instructors the ability to benefit 
from ongoing feedback (see McKone, 1999:407) in an atmosphere unsullied from 
faculty's concerns around tenure processes and salary decisions (see Chen & Hoshower, 
1998). The following research was intended to address the limitations of traditional 
student assessments as well as to assess the effectiveness of an independent, faculty- 
administered online tool allowing teachers to benefit from anonymous student input 
without formal summative repercussions. 

Anonymous Online Assessment Pilot Research 

In January 2000, five instructors from Mount Royal College volunteered to participate in 
a study investigating the usefulness of an anonymous online faculty-administered 
course/teacher assessment form. The faculty, and the courses they selected for study, was 
a diverse group that provided a unique opportunity to evaluate the online assessment 
software from a cross-disciplinary perspective (see Table 1). 



Table 1. Discipline/Class Participant 


ts 


Discipline 


Class 


Number of Students 


Biology 


Introduction to Biology (first year) 


49 


Interior Design 


Computer Drafting (second year) 


40 


Physics 


Classical Physics II (first year) 


31 


Socioiogy 


Introduction to Sociology (first year) 


33 


Theatre 


Theatre Research (first year) 


16 



Faculty were required to attend 2 one-hour focus group discussions (in February and 
April^) as well as volunteer one of their classes to use the online assessment. Once the 
classes were selected they were visited to provide students with an overview of the 
research and to answer any questions. All students wishing to participate were required 
to sign consent forms reiterating the parameters and purpose of the research. While all 
students in the class were able to complete the anonymous assessment, random selections 
of 5 students from each class were asked to participate in the focus groups. No financial 
remuneration for participating in the focus groups was provided to students or faculty. 
The total number of focus group participants was 30 (i.e., 5 faculty and 25 students). 

All focus group discussions were guided by, but not limited to, answering the following 3 
questions: 

1. Was the assessment tool easy to use? 

2. Were the results from the assessment useful for improving teaching? 

3. Did using the assessment tool inspire faculty and students to view the course and/or 
teaching from a new perspective? 



^ Conducting focus group discussion at the begiiming of the course and at the end allows 
an investigation of how the assessment tool was used over the course and what, if any, 
influence it had on learning and teaching. 
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Attendance in the February focus groups was 100% for both students and faculty. This is 
extraordinary when one considers that these meetings were held outside of class time and 
required coordinating 5 people's busy schedules. The April sessions included 100% 
attendance for faculty and 60% for students (for an overall attendance rate of 83%). 
Students attending the second series of sessions highlighted the fact that the meetings 
were scheduled immediately before exams and that this may have accounted for the 
lower attendance. 

To determine the usefulness of the online assessment from both faculty and student 
perspectives, a brief review of the quantitative and qualitative results from the pilot 
project should prove useful. 



Quantitative Resuits 

One criterion to determine if the online tool was useful would be the number of 
assessments students submitted. 



Table 2. Quantitative Results 




Discipline 


Students 


Number of Assessments 


Total 


Class^ 






1 


2 


3. 


4 






Biology 


49 


24 


5 






29 


59% 


Interior Design 


40 


4 


1 






5 


13% 


Physics ^ 


31 1 


12 


2 


2 




16 ^ 


52% 


Sociology 


33 


13 


5 


1 




19 


58% 


Theatre 


16 


6 


1 


1 


1 


9 


56% 


Total 


169 


59 


14 


4 


1 


1 78 


48% 



Table 2 summarizes the number of online assessments students submitted. One would 
predict, given Chen and Hoshower's (1998) research indicating that students are more 
motivated to participate in teaching evaluations when the results are intended to improve 
teaching, that students would be highly motivated to complete the survey. This is based 
on the fact that in the pilot the instructors were only using the assessment's results to 
improve their teaching - there were no tenure/pay raise implications. However, aggregate 
results show that fewer than 35% of students (i.e., 59/169) completed the online 
assessment at least once during the term. Although this appears to challenge Chen and 
Hoshower's findings, focus group discussions revealed some compelling reasons why 
students did not submit more assessments. 

Qualitative Results 

Interpreting the relatively low numbers of assessments students submitted highlights two 
interesting findings from this research. Student’s Approach to Teaching Assessment and 
faculty's Numerical Fixation. 



On the survey, each student was asked to indicate how many times he or she had 
submitted an assessment (i.e., 1-4 times). 

^ Since the software is anonymous these percentages are illustrative only and may not 
accurately reflect the proportion of students who submitted an assessment. 
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Student's Approach to Teaching Assessment 

During the focus groups students were quick to point out that they liked their teachers 
and thought they were doing a good job. The facilitator's final report suggests that 
students expressed the belief that if they were content with their teacher's performance, 
there was no reason to complete the survey. Therefore, the lack of student participation 
may be an indication that the teacher was doing a good job, not the reverse. 

I 

The Interior Design instructor, for example, who had the lowest number/proportion of 
assessments submitted, was clearly adored by her students as demonstrated in the focus 
groups. When the facilitator asked her students why they were not submitting more 
assessments, the students said they liked the course and that if they had concerns they felt 
they could approach her directly and "tell her to her face". For this class then, students 
felt they were receiving quality instruction and did not feel they needed to complete an 
assessment. 

Numerical Fixation 

As faculty, our fixation on numbers (i.e., we attributed good teaching with the number of 
assessments submitted, or as we referred to them, "hits") was evident during our focus 
group discussions as well as our informal meetings in the halls. In the focus group some 
faculty expressed that they did not know why their students were not submitting more 
assessments. As discussed above, faculty were equating the number of assessments with 
their teaching performance and this interpretation may have been misguided. 

These qualitative findings add support for the position that even though student 
participation rates were low, that this should not diminish the value of providing the 
students with access to the tool. One student mentioned that even though she had not 
completed the online assessment, "it was nice to know that it was there". 

To review the results of the focus groups, and their discussions of the online assessment 
tool and its influence on teaching, a summary of the positive and negative impressions 
from both student and faculty perspectives should prove valuable. 

Student Impressions (positive): 

Students felt that using the online assessment was a positive experience in a number of 
ways. 

First, students found the assessment tool easy to use and straightforward to understand. 
The assessment was easy to find because it was linked directly from the course's web- 
page. Students stated that navigating to the survey, completing and submitting it, was a 
clear and straightforward process. 

Second, students liked that the survey was anonymous. Some students in the focus 
groups mentioned that when completing print-based assessments they were conscious of 
whether or not the instructor could attribute individual comments to them. This finding 
supports the potential value of using an anonymous online teaching assessment (see 
Feldman, 1979; Blunt, 1991; CAUT, 1998 on issues surrounding student anonymity). 
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Third, students enjoyed commenting on a class while they were still in it. Students 
highlighted that commenting on their course during the term was a better assessment 
strategy than were traditional assessments conducted at the end of the class. While 
students acknowledged the need for more formal evaluations (i.e., those conducted by the 
administration), they were unsure if their comments on these assessments ever went 
anywhere, and if they did, that the resulting changes would never benefit them. This 
reinforces the position taken by McKone (1999:407) who recommends that instructors 
establish mechanisms for ongoing feedback to students throughout the term. 

Fourth, students liked having the tool available even if they did not use it. As mentioned 
earlier, students felt that having the option of completing the assessment was important 
even if they chose not to submit one. 

Fifth, students felt that the online assessment allowed them to offer more thoughtful 
comments than did the traditional, in-class, print-based teaching evaluations. Because the 
online form is available 24 hours a day students could complete them when they wanted 
to, not when they had to. Students suggested that the in-class assessments did not allow 
them the flexibility that the online version did. 

Sixth, students appreciated knowing that their teacher wanted to improve their teaching 
and/or their course. Students commented that they liked knowing that their teacher 
wanted to improve and that this made them appear more human and approachable. 

Student Impressions (areas for improvement/concern): 

Students felt that using the online assessment had some drawbacks. These criticisms 
focused on 4 issues. 

First, students were concerned about access to computers. Some students pointed out that 
access to the survey was limited to those students who had computers at home even 
though Mount Royal College provides student's with access to computers that are 
available at various times and locations. When questioned about this, students suggested 
that the computers at school were slow and that lab times were not always convenient. 

Second, some students had difficulty remembering the password. Passwords for the 
software were presented in class but students said that it was often inconvenient for them 
to find the password when they wanted to submit an assessment. 

Third, some students questioned if their teacher looked at the assessments. Some focus 
groups suggested that their instructor rarely, if ever, discussed the results of the surveys 
in class. This criticism however, was not true of all instructors and the majority of focus 
groups said that their instructor did refer to the survey in class. 

Fourth, students reinforced that the questions needed to be changed so they remained 
current and pertinent. The software available during the pilot research did not allow the 
instructor to change/edit questions during the term. By default then, the questions 
focused on the entire course and as such, student comments may not have changed much 
from one point in the course to the next. 
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Faculty Impressions (positive): 

Faculty suggested that there were some clear benefits to participating in the pilot research 
itself and using the online assessment tool. 

First, faculty enjoyed participating in an open dialogue about teaching with their 
colleagues. Although all participating faculty had vast teaching experience they 
mentioned how rarely they actually discussed teaching. Even though focus group 
sessions were dedicated at assessing the use of the online tool, discussions often 
transcended the software and became a lively and invigorating discussion about teaching 
and learning. Faculty expressed that these discussions were among the most beneficial 
aspects of participating in the pilot research. 

Second, the assessment tool made teaching a more reflective process. In fact, one faculty 
member suggested that because the online assessment tool was available to students 24 
hours a day, every day, he found himself being more prepared for class. He saw this as a 
good thing and that knowing he could be assessed after every class made him more 
conscious of his teaching and how he might try to improve. 

Third, the results from the assessments helped the teacher address student concerns in a 
proactive and constructive manner. For those faculty who discussed the results from the 
assessment with their classes (3 of 5) suggested that addressing student concerns in this 
way allowed them to respond in a more thoughtful manner. Instead of responding to 
student concerns on their feet instructors could reflect on the online comments and then 
raise the issue/concem with the rest of the class when they were prepared. Once in class, 
the instructor could ask the students if the concern raised online was important to them as 
well, and if it was, how the teacher planned (or didn't plan) to address it. 

Fourth, the assessment tool allowed faculty to demonstrate that they practiced what they 
preached. One faculty member mentioned that the online tool allowed him to reinforce 
his belief that all teachers, he included, had much to learn from his student's honest and 
sincere criticism. He also suggested that as teachers we continually provide students with 
feedback on their performance (i.e., comments on tests, papers, etc.,) but that we rarely 
give ourselves the same opportunity for improvement. The online assessment allowed 
faculty to reinforce their desire to become better teachers through an active dialogue with 
their students. 

Faculty Impressions (areas for improvement/concern): 

Faculty did not voice any damning criticisms of the online assessment initiative but did 
offer some specific comments on how the software could be improved, and some general 
reflections on changing the existing teaching assessment culture. 

First, to be an effective and proactive tool for improving teaching the online software 
must allow the instructor to alter their questions when they want. Comments gained from 
static questions do not allow the student or the instructor the benefit to assess the 
changing needs in the classroom. Questions need to be dynamic in order to inspire 
students to submit the assessment more than once, and more responsive to individual 
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courses so that they can account for individual coiu'se needs (i.e., questions about labs vs. 
lectures, guest presentations vs. field trips, etc.). 

Second, faculty should not associate the number of assessments with their popularity 
and/or teaching ability. As mentioned previously, future users of the software must 
realize that the number of assessments does not necessarily correspond to how well, or 
how poorly, they are doing. Faculty reinforced the belief that the online assessment 
should be only one of many vehicles the instructor should use to determine how the 
students are doing in the class and when changes to delivery or curricula are required. 

Third, faculty should continually reinforce the availability of the assessment tool in class. 
Two of the 5 faculty stated that they should have referred to and reinforced the 
availability of the tool in class. Both said that this was not necessarily to get more hits 
per s6, but to let students know that they wanted to hear their comments on how they 
could improve their teaching and the course. 

Fourth, faculty said they wanted to change the teaching assessment culture for students 
and faculty. Faculty stated that both our students and our colleagues need to reflect more 
on the teaching assessment process and by doing so, change the assessment culture to one 
that is more dynamic and constructive. On the one hand, students need to understand that 
their instructors can benefit from hearing negative, constructive criticisms, as well as 
hearing that they enjoy the course and look forward to coming to class. On the other 
hand, faculty need to appreciate the wealth of knowledge that our students have about 
teaching and that we should allow ourselves to benefit from it. Teaching assessments 
need not inspire hostility and cynicism among faculty but instead be seen as one vehicle 
by which we can improve the classroom for both the student and the teacher. 

Analysis and Discussion 

Given the previous review, a return to the pilot's research questions is necessary to 
determine the value of the online assessment tool. 

1 . Did students and faculty find the software easy to use? 

Yes, students found that the online assessment was easy to complete and submit. Faculty 
found the tool easy to use and the software's reporting features excellent. 

2. Were the results from the assessment useful for improving teaching? 

Partly. Although more research on how the online assessment tool influences teaching is 
required, preliminary results indicate that the ongoing feedback the tool provides to 
instructors may be invaluable for refining teaching techniques and/or curricula. 

3. Did using the assessment tool inspire faculty and students to view the course and/or 
teaching from a new perspective? 

Potentially. For 3 of the 5 faculty the results from the online assessments were discussed 
in class. For them, this level of classroom discussion about teaching was a novel and 
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unique experience. Faculty comments indicate that they felt this was a useful mechanism 
for interacting with their students about their teaching philosophy and their goals for the 
course. Students in these classes mentioned that when the instructor opened the 
discussion about their teaching that it made them feel more involved in the teaching and 
learning process. Although more research is required, it appears that when instructors 
used the results from the assessment tool and discussed them with their students, they 
were rewarded by increased instructor/student interaction. 

Although the anonymous online assessment software is in its infancy the tool holds great 
promise for allowing instructors to benefit from student comments during the course. 
Certainly, more work needs to be done, but so far, the benefits of the software far 
outweigh its cost in time and effort.^ 



® There are no costs for instructors to use this software. The software is available at: 
http://zpatz.mtroyal.ab.ca/fast/ 
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